Task based evaluation of exploratory search systems

Evaluation of interactive search systems has always been time-consuming and complex, which probably explains the relative low level of interest from IR researchers for this type of evaluation in the past. Yet the limitations of batch-style system evaluations cannot be ignored anymore. We present some case studies of evaluations in interactive settings. Several of these evaluations offer valuable new insights about system adequacy. This more than compensates for the reduced ability to reproduce results. We distinguish system centered evaluations focusing on performance and user centered (task based) evaluations focusing on adequacy. The latter take the natural task of a user as starting point. Task based evaluations suggest that proper HCI design is probably a more important factor for user satisfaction than the quality of statistical indexing and ranking methods. User centered and system centered evaluations of interactive systems measure different aspects of quality. The challenge is to design an evaluation where the different components that determine system adequacy and performance can be identified and their relationship can be quantified.